Panoramic camera, which captures the 360° view of scenic places such as tourist resorts, is an example of a panoramic imaging system. Panoramic imaging system is an imaging system that captures the views one could get by making one complete turn-around from a given spot. On the other hand, an omnidirectional imaging system captures the views of every possible direction from a given position. Omnidirectional imaging system provides a view that a person could observe from a given position by turning around as well as looking up and down. In a mathematical terminology, the solid angle of the region that can be captured by the imaging system is 4π steradian.
There have been a lot of studies and developments of panoramic and omnidirectional imaging systems not only in the traditional areas such as photographing buildings, nature scenes, and heavenly bodies, but also in security/surveillance systems using CCD (charge-coupled device) or CMOS (complementary metal-oxide-semiconductor) cameras, virtual touring of real estates, hotels and tourist resort, and navigational aids for mobile robots and unmanned aerial vehicles(UAV).
One method of obtaining a panoramic image is to employ a fisheye lens with a wide field of view(FOV). For example, the stars in the entire sky and the horizon can be captured in a single image by pointing a camera equipped with a fisheye lens with 180° FOV toward the zenith (i.e., the optical axis of the camera is aligned perpendicular to the ground plane). On this reason, fisheye lenses have been often referred to as “all-sky lenses”. Particularly, a high-end fisheye lens by Nikon, namely, 6 mm f/5.6 Fisheye-Nikkor, has a FOV of 220°. Therefore, a camera equipped with this lens can capture a portion of the backside of the camera as well as the front side of the camera. Then, panoramic image can be obtained from thus obtained fisheye image after proper image processing.
In many cases, imaging systems are installed on vertical walls. Imaging systems installed on outside walls of a building for the purpose of monitoring the surroundings, or a rear view camera for monitoring the backside of a passenger car are such examples. In such cases, it is rather inefficient if the horizontal field of view is significantly larger than 180°. This is because a wall, which is not needed to be monitored, takes up a large space in the monitor screen. Pixels are wasted in this case, and the screen appears dull. Therefore, a horizontal FOV around 180° is more appropriate for such cases. Nevertheless, a fisheye lens with 180° FOV is not desirable for such applications. This is because the barrel distortion, which accompanies a fisheye lens, evokes psychological discomfort and thus abhorred by the consumer.
References 1 and 2 provide fundamental technologies of extracting an image having a particular viewpoint or projection scheme from an image having a different viewpoint or projection scheme. Specifically, reference 2 provides an example of a cubic panorama. In short, a cubic panorama is a special technique of illustration wherein the observer is assumed to be located at the very center of an imaginary cubic room made of glass, and the outside view from the center of the glass room is directly transcribed on the region of the glass wall where the ray vector from the object to the observer meets the glass wall. However, the environment was not a real environment captured by an optical lens, but a computer-created imaginary environment captured with an imaginary distortion-free pinhole camera.
From another point of view, all animals and plants including human are bound on the surface of the earth due to the gravitational pull, and most of the events, which need attention or cautionary measure, take place near the horizon. Therefore, even though it is necessary to monitor every 360° direction on the horizon, it is not as important to monitor high along the vertical direction, for example, as high as to the zenith or deep down to the nadir. Distortion is unavoidable if we want to describe the scene of every 360° direction on a two-dimensional plane. Similar difficulty exists in the cartography where geography on Earth, which is a structure on the surface of a sphere, needs to be mapped on a planar two-dimensional atlas.
All the animals, plants and inanimate objects such as buildings on the Earth are under the influence of gravity, and the direction of the gravitational force is the up-right direction or the vertical direction. Among all the distortions, the distortion that appears most unnatural to the human is the distortion where vertical lines appear as curved lines. Therefore, even if other kinds of distortions are present, it is important to make sure that such a distortion is absent. In general, a ground plane is perpendicular to the direction of the gravitational force, but it not so on a slanted ground. Therefore, in a strict sense of the words, a reference has to be made with respect to the horizontal plane, and a vertical direction is a direction perpendicular to the horizontal plane.
Described in reference 3 are the well-known map projection schemes among the diverse map projection schemes such as equi-rectangular projection, Mercator projection and cylindrical projection schemes, and reference 4 provides a brief history of diverse map projection schemes. Among these, the equi-rectangular projection scheme is the projection scheme most familiar to us when we describe the geography on the Earth, or when we draw the celestial sphere in order to make a map of the constellation.
Referring to FIG. 1, if we assume the surface of the Earth or the celestial sphere is a spherical surface with a radius S, then an arbitrary point Q on the Earth's surface with respect to the center N of the Earth has a longitude ψ and a latitude δ. On the other hand, FIG. 2 is a schematic diagram of a planar map of the Earth's surface or the celestial sphere drawn according to the equi-rectangular projection scheme. A point Q on the Earth's surface having a longitude ψ and a latitude δ has a corresponding point P″ on the planar map(235) drawn according to the equi-rectangular projection scheme. The rectangular coordinate of this corresponding point is given as (x″, y″). Furthermore, the reference point on the equator having a longitude 0° and a latitude 0° has a corresponding point ″ on the planar map, and this corresponding point ″ is the origin of the rectangular coordinate system. Here, according to the equi-rectangular projection scheme, a same interval in the longitude (i.e., a same angular distance along the equatorial line) corresponds to a same lateral interval on the planar map. In other words, the lateral coordinate x″ on the planar map(235) is proportional to the longitude.x″=cψ  [Equation 1]
Here, c is proportionality constant. Also, the longitudinal coordinate y″ is proportional to the latitude, and has the same proportionality constant as the lateral coordinate.y″=cδ  [Equation 2]
Such an equi-rectangular projection scheme appears as a natural projection scheme considering the fact that the Earth's surface is close to a sphere. Nevertheless, it is disadvantageous in that the size of a geographical area is greatly distorted. For example, two very close points near the North Pole can appear as if they are on the opposite sides of the Earth in a map drawn according to the equi-rectangular projection scheme.
On the other hand, in a map drawn according to the Mercator projection scheme, the longitudinal coordinate is given as a complex function given in Eq. 3.
                              y          ″                =                  c          ⁢                                          ⁢          ln          ⁢                      {                          tan              ⁡                              (                                                      π                    4                                    +                                      δ                    2                                                  )                                      }                                              [                  Equation          ⁢                                          ⁢          3                ]            
On the other hand, FIG. 3 is a conceptual drawing illustrating a cylindrical projection scheme or a panoramic perspective. In a cylindrical projection scheme, an imaginary observer is located at the center N of a celestial sphere(331) with a radius S, and it is desired to make a map of the celestial sphere centered on the observer, the map covering most of the region, but excluding the zenith and the nadir. In other words, the span of the longitude must be 360° ranging from −180° to +180°, but the range of the latitude can be narrower including the equator within its span. Specifically, the span of the latitude can be assumed as ranging from −Δ to +Δ, and here, Δ must be smaller than 90°.
In this projection scheme, an imaginary cylindrical plane(334) is assumed which contacts the celestial sphere(331) at the equator(303). Then, for a point Q(ψ, δ) on the celestial sphere having a given longitude ψ and a latitude δ, a line segment connecting the center N of the celestial sphere and the point Q is extended until it meets the said cylindrical plane. This intersection point is designated as P(ψ, δ). In this manner, a corresponding point P on the cylindrical plane(334) can be obtained for every point Q on the celestial sphere(331) within the said latitude range. Then, a map having a cylindrical projection scheme is obtained when the cylindrical plane is cut and flattened out. Therefore, the lateral coordinate x″ of the point P on the flattened-out cylindrical plane is given by Eq. 4, and the longitudinal coordinate y″ is given by Eq. 5.x″=Sψ  [Equation 4]y″=S tan δ  [Equation 5]
Such a cylindrical projection scheme is used in panoramic cameras that produce panoramic images by rotating in the horizontal plane. Especially, if the lens mounted on the rotating panoramic camera is a distortion-free rectilinear lens, then the resulting panoramic image exactly follows a cylindrical projection scheme. In principle, such a cylindrical projection scheme is the most accurate panoramic projection scheme. However, the panoramic image appears unnatural when the latitudinal range is large, and thus it is not widely used in practice.
References 5 and 6 provide an example of a fisheye lens with 190° FOV, and reference 7 provides various examples of wide-angle lenses including dioptric and catadioptric fisheye lenses with stereographic projection schemes.
On the other hand, reference 8 provides various examples of obtaining panoramic images following cylindrical projection schemes, equi-rectangular projection schemes, or Mercator projection schemes from images acquired using rotationally symmetric wide-angle lenses including fisheye lenses. Referring to FIG. 4 through FIG. 12, most of the examples provided in the said references can be summarized as follows.
FIG. 4 is a conceptual drawing illustrating a real projection scheme of a rotationally symmetric wide-angle lens 412) including a fisheye lens. Z-axis of the world coordinate system describing objects captured by the wide-angle lens coincides with the optical axis(401) of the wide-angle lens(412). An incident ray(405) having a zenith angle θ with respect to the Z-axis is refracted by the lens(412), and as a refracted ray(406), converges toward an image point P on the focal plane(432). The distance between the nodal point N of the lens and the said focal plane is approximately equal to the effective focal length of the lens. The sub area on the focal plane whereon real image points have been formed is the image plane(433). To obtain a sharp image, the said image plane(433) must coincide with the image sensor plane(413) within the camera body(414). Said focal plane and the said image sensor plane are perpendicular to the optical axis. The intersection point  between the optical axis(401) and the image plane(433) is hereinafter referred to as the first intersection point. The distance between the first intersection point and the said image point P is r.
For general wide-angle lenses, the image height r is given by Eq. 6.r=r(θ)  [Equation 6]
Here, the unit of the incidence angle θ is radian, and the above function r(θ) is a monotonically increasing function of the zenith angle θ of the incident ray.
Such a real projection scheme of a lens can be experimentally measured using an actual lens, or can be calculated from the lens prescription using dedicated lens design software such as Code V or Zemax. For example, the y-axis coordinate y of the image point on the focal plane by an incident ray having given horizontal and vertical incidence angles can be calculated using a Zemax operator REAY, and the x-axis coordinate x can be similarly calculated using an operator REAX.
FIG. 5 is an imaginary interior scene produced by Professor Paul Bourke by using a computer, and it has been assumed that an imaginary fisheye lens with 180° FOV and having an ideal equidistance projection scheme has been used to capture the scene. This image is a square image, of which both the lateral and the longitudinal dimensions are 250 pixels. Therefore, the coordinate of the optical axis is (125.5, 125.5), and the image height for an incident ray with a zenith angle of 90° is given as r′(π/2)=125.5−1=124.5. Here, r′ is not a physical distance, but an image height measured in pixel distance. Since this imaginary fisheye lens follows an equidistance projection scheme, the projection scheme of this lens is given by Eq. 7.
                                          r            ′                    ⁡                      (            θ            )                          =                                            124.5                              (                                  π                  2                                )                                      ⁢            θ                    =                      79.26            ⁢            θ                                              [                  Equation          ⁢                                          ⁢          7                ]            
FIG. 6 through FIG. 8 show diverse embodiments of wide-angle lenses presented in reference 7. FIG. 6 is a dioptric(i.e., refractive) fisheye lens with a stereographic projection scheme, FIG. 7 is a catadioptric fisheye lens with a stereographic projection scheme, and FIG. 8 is a catadioptric panoramic lens with a rectilinear projection scheme. In this manner, wide-angle lenses from the prior arts and in the current invention are not limited to fisheye lenses with equidistance projection schemes, but encompass all kind of wide-angle lenses that are rotationally symmetric about the optical axes.
The main point of the invention in reference 8 is about providing methods of obtaining panoramic images by applying mathematically accurate image processing algorithms on images obtained using rotationally symmetric wide-angle lenses. Referring to FIG. 9, diverse embodiments in reference 8 can be summarized as follows. FIG. 9 is a schematic diagram of a world coordinate system of prior arts.
The world coordinate system of the said invention takes the nodal point N of a rotationally symmetric wide-angle lens as the origin, and a vertical line passing through the origin as the Y-axis. Here, the vertical line is a line perpendicular to the ground plane, or more precisely to the horizontal plane(917). The X-axis and the Z-axis of the world coordinate system are contained within the ground plane. The optical axis(901) of the said wide-angle lens generally does not coincide with the Y-axis, and can be contained within the ground plane(i.e., parallel to the ground plane), or may not contained within the ground plane. The plane(904) containing both the said Y-axis and the said optical axis(901) is referred to as the reference plane. The intersection line(902) between this reference plane(904) and the ground plane(917) coincides with the Z-axis of the world coordinate system. On the other hand, an incident ray(905) originating from an object point Q having a rectangular coordinate (X, Y, Z) in the world coordinate system has an altitude angle δ from the ground plane, and an azimuth angle ψ with respect to the reference plane. The plane(906) containing both the Y-axis and the said incident ray(905) is the incidence plane. The horizontal incidence angle ψ of the said incident ray with respect to the said reference plane is given by Eq. 8.
                    ψ        =                              tan                          -              1                                ⁡                      (                          X              Z                        )                                              [                  Equation          ⁢                                          ⁢          8                ]            
On the other hand, the vertical incidence angle (i.e., the altitude angle) δ subtended by the said incident ray and the X-Z plane is given by Eq. 9.
                    δ        =                              tan                          -              1                                ⁡                      (                          Y                                                                    X                    2                                    +                                      Z                    2                                                                        )                                              [                  Equation          ⁢                                          ⁢          9                ]            
The elevation angle μ of the said incident ray is given by Eq. 10, wherein χ is an arbitrary angle larger than −90° and smaller than 90°.μ=δ−χ[Equation 10]
FIG. 10 is a schematic diagram of a panoramic imaging system, and mainly comprised of an image acquisition means(1010), an image processing means(1016) and image display means(1015, 1017). The image acquisition means(1010) includes a rotationally symmetric wide-angle lens(1012) and a camera body(1014) having an image sensor(1013) inside. The said wide-angle lens can be a fisheye lens with more than 180° FOV and having an equidistance projection scheme, but it is by no means limited to such a fisheye lens. Rather, it can be any rotationally symmetric wide-angle lens including a catadioptric fisheye lens. Hereinafter, for the sake of notational simplicity, a wide-angle lens will be referred to as a fisheye lens. Said camera body contains image sensors such as CCD or CMOS sensors, and it can acquire either a still image or a movie. By the said fisheye lens(1012), a real image of the object plane(1031) is formed on the focal plane(1032). In order to obtain a sharp image, the image sensor plane(1013) must coincide with the focal plane(1032). In FIG. 10, the symbol 1001 refers to the optical axis.
The real image of the objects on the object plane(1031) formed by the fisheye lens(1012) is converted by the image sensor(1013) into electrical signals, and displayed as an uncorrected image plane(1034) on an image display means(1015). This uncorrected image plane(1034) is a distorted image by the fisheye lens. This distorted image plane can be rectified by the image processing means(1016), and then displayed as a processed image plane(1035) on an image display means(1017) such as a computer monitor or a CCTV monitor. Said image processing can be accomplished by software running on a computer, or it can be done by embedded software running on FPGA(Field Programmable Gate Array), CPLD(Complex Programmable Logic Device), ARM core processor, ASIC(Application-Specific Integrated Circuit), or DSP(Digital Signal Processor) chips.
An arbitrary rotationally symmetric lens including a fisheye lens does not provide said panoramic images or distortion-free rectilinear images. Therefore, image processing stage is essential in order to obtain desirable images. FIG. 11 is a conceptual drawing of an uncorrected image plane(1134) prior to an image processing stage, which corresponds to the real image on the image sensor plane(1013). If the lateral dimension of the image sensor plane(1013) is B and the longitudinal dimension is V, then the lateral dimension of the uncorrected image plane is gB and the longitudinal dimension is gV, where g is proportionality constant.
Uncorrected image plane(1134) can be considered as an image displayed on an image display means without rectification of distortion, and is a magnified image of the real image on the image sensor plane by a magnification ratio g. For example, the image sensor plane of a ⅓-inch CCD sensor has a rectangular shape having a lateral dimension of 4.8 mm, and a longitudinal dimension of 3.6 mm. On the other hand, if the monitor is 48 cm in width and 36 cm in height, then the magnification ratio g is 100. More desirably, the side dimension of a pixel in digital image is considered as 1. A VGA-grade ⅓-inch CCD sensor has pixels in an array form with 640 columns and 480 rows. Therefore, each pixel has a right rectangular shape with both width and height measuring as 4.8 mm/640=7.5 μm, and in this case, the magnification ratio g is given by 1 pixel/7.5 μm=133.3 pixel/mm. In recapitulation, the uncorrected image plane(1134) is a distorted digital image obtained by converting the real image formed on the image sensor plane into electrical signals.
The first intersection point  on the image sensor plane is the intersection point between the optical axis and the image sensor plane. Therefore, a ray entered along the optical axis forms an image point on the said first intersection point . By definition, the point ′ on the uncorrected image plane corresponding to the first intersection point  on the image sensor plane—hereinafter referred to as the second intersection point—corresponds to an image point by an incident ray entered along the optical axis.
A second rectangular coordinate systems is assumed wherein x′-axis is taken as the axis that passes through the second intersection point ′ on the uncorrected image plane and is parallel to the sides of the uncorrected image plane along the lateral direction, and y′-axis is taken as the axis that passes through the said second intersection point and is parallel to the sides of the uncorrected image plane along the longitudinal direction. The positive direction of the x′-axis runs from the left side to the right side, and the positive direction of the y′-axis runs from the top end to the bottom end. Then, the lateral coordinate x′ of an arbitrary point on the uncorrected image plane(1134) has a minimum value x′1=gx1 and a maximum value x′2=gx2(i.e., gx1≦x′≦gx2). In the same manner, the longitudinal coordinate y′ of the said point has a minimum value y′1=gy1 and a maximum value y′2=gy2(i.e., gy1≦y′≦gy2).
FIG. 12 is a conceptual drawing of a processed image plane(1235) of the image display means of the current invention, wherein the distortion has been removed. The processed image plane(1235) has a rectangular shape, of which the lateral side measuring as W and the longitudinal side measuring as H. Furthermore, a third rectangular coordinate system is assumed wherein x″-axis is parallel to the sides of the processed image plane along the lateral direction, and y″-axis is parallel to the sides of the processed image plane along the longitudinal direction. The direction of the z″-axis of the third rectangular coordinate system coincides with the direction of the z-axis of the first rectangular coordinate system and the direction of the z′-axis of the second rectangular coordinate system. The intersection point ″ between the said z″-axis and the processed image plane—hereinafter referred to as the third intersection point—can take an arbitrary position, and it can even be located outside the processed image plane. Here, the positive direction of the x″-axis runs from the left side to the right side, and the positive direction of the y″-axis runs from the top end to the bottom end.
The first and the second intersection points correspond to the location of the optical axis. On the other hand, the third intersection point corresponds not to the location of the optical axis but to the principal direction of vision. The principal direction of vision may coincide with the optical axis, but it is not needed to. Principal direction of vision is the direction of the optical axis of an imaginary panoramic or rectilinear camera corresponding to the desired panoramic or rectilinear images. Hereinafter, for the sake of notational simplicity, the principal direction of vision is referred to as the optical axis direction.
The lateral coordinate x″ of a third point P″ on the processed image plane(1235) has a minimum value x″1 and a maximum value x″2(i.e., x″1≦x″≦x″2). By definition, the difference between the maximum lateral coordinate and the minimum lateral coordinate is the lateral dimension of the processed image plane(i.e., x″2−x″1=W). In the same manner, the longitudinal coordinate y″ of the third point P″ has a minimum value y″1 and a maximum value y″2(i.e., y″1≦y″≦y″2). By definition, the difference between the maximum longitudinal coordinate and the minimum longitudinal coordinate is the longitudinal dimension of the processed image plane(i.e., y″2−y″1=H).
The following table 1 summarizes corresponding variables in the object plane, the image sensor plane, the uncorrected image plane, and the processed image plane.
TABLE 1imageuncorrected imageprocessed imageSurfaceobject planesensor planeplaneplanelateral dimension BgBWof the planelongitudinal dimensionVgVHof the planecoordinate systemworld the first the secondthe third coordinaterectangularrectangularrectangularsystemcoordinatecoordinatecoordinatesystemsystemlocation of the nodal point nodal point of nodal point of nodal point of coordinate originof the lensthe lensthe lensthe lenssymbol of the originOO′O″coordinate axes(X, Y, Z)(x, y, z)(x′, y′, z′)(x″, y″, z″)name of the object object pointthe first pointthe second pointthe third pointpoint or the image pointsymbol of the object pointQPP′P″or the image pointtwo-dimensional (x, y)(x′, y′)(x″, y″)coordinateof the object pointor the image point
On the other hand, if we assume the coordinate of an image point P″ on the processed image plane(1235) corresponding to an object point with a coordinate (X, Y, Z) in the world coordinate system is (x″, y″), then the said image processing means process the uncorrected image plane so that an image point corresponding to an incident ray originating from the said object point appears on the said screen with the coordinate (x″, y″), wherein the lateral coordinate x″ of the image point is given by Eq. 11.x″=cψ  [Equation 11]
Here, c is proportionality constant.
Furthermore, the longitudinal coordinate y″ of the said image point is given by Eq. 12.y″=cF(μ)  [Equation 12]
Here, F(μ) is a monotonically increasing function passing through the origin. In mathematical terminology, it means that Eq. 13 and Eq. 14 are satisfied.
                              F          ⁡                      (            0            )                          =        0                            [                  Equation          ⁢                                          ⁢          13                ]                                                      ∂                          F              ⁡                              (                μ                )                                                          ∂            μ                          >        0                            [                  Equation          ⁢                                          ⁢          14                ]            
The above function F(μ) can take an arbitrary form, but the most desirable forms are given by Eq. 15 through Eq. 18.
                              F          ⁡                      (            μ            )                          =                  tan          ⁢                                          ⁢          μ                                    [                  Equation          ⁢                                          ⁢          15                ]                                          F          ⁡                      (            μ            )                          =                              tan            ⁢                                                  ⁢            μ                                cos            ⁢                                                  ⁢            χ                                              [                  Equation          ⁢                                          ⁢          16                ]                                          F          ⁡                      (            μ            )                          =        μ                            [                  Equation          ⁢                                          ⁢          17                ]                                          F          ⁡                      (            μ            )                          =                  ln          ⁢                      {                          tan              ⁡                              (                                                      μ                    2                                    +                                      π                    4                                                  )                                      }                                              [                  Equation          ⁢                                          ⁢          18                ]            
FIG. 13 is a schematic diagram for understanding the field of view(FOV) and the projection scheme of a panoramic imaging system according to an embodiment of a prior art. A panoramic imaging system of the current embodiment is assumed as attached on a vertical wall(1330), which is perpendicular to the ground plane. The wall coincides with the X-Y plane, and the Y-axis runs from the ground plane(i.e., X-Z plane) to the zenith. The origin of the coordinate system is located at the nodal point N of the lens, and the optical axis(1301) of the lens coincides with the Z-axis. In a rigorous sense, the direction of the optical axis is the direction of the negative Z-axis of the world coordinate system. This is because, by the notational convention of imaging optics, the direction from the object(or, an object point) to the image plane(or, an image point) is the positive direction. Despite this fact, we will describe the optical axis as coinciding with the Z-axis of the world coordinate system for the sake of simplicity in argument. The present invention is not an invention about designing a lens but an invention employing a lens, and in the viewpoint of a lens user, it makes easier to understand the optical axis as in the current embodiment of the present invention.
The image sensor plane(1313) is a plane having a rectangular shape and perpendicular to the optical axis, whereof the lateral dimension is B, and the longitudinal dimension is V. In the current embodiment, the X-axis of the world coordinate system is parallel to the x-axis of the first rectangular coordinate system, and points in the same direction. On the other hand, the Y-axis of the world coordinate system is parallel to the y-axis of the first rectangular coordinate system, but the direction of the Y-axis is the exact opposite of the direction of the y-axis. Therefore, in FIG. 13, the positive direction of the x-axis of the first rectangular coordinate system runs from the left to the right, and the positive direction of the y-axis runs from the top to the bottom.
The intersection point  between the z-axis of the first rectangular coordinate system and the sensor plane(1313)—in other words, the first intersection point—is not generally located at the center of the sensor plane(1313), and it can even be located outside the sensor plane. Such a case can happen when the center of the image sensor is moved away from the center position of the lens—i.e., the optical axis—on purpose in order to obtain an asymmetrical vertical or horizontal field of view.
A panoramic camera with a cylindrical projection scheme follows a rectilinear projection scheme in the vertical direction, and an equidistance projection scheme in the horizontal direction. Such a projection scheme corresponds to assuming a hemi-cylindrical object plane(1331) with a radius S and having the Y-axis as the rotational symmetry axis. The image of an arbitrary point Q on the object plane(1331)—hereinafter referred to as an object point—appears as an image point P on the said sensor plane(1313). According to the desirable projection scheme of the current embodiment, the image of an object on the hemi-cylindrical object plane(1331) is captured on the sensor plane(1313) with its vertical proportions preserved, and the lateral coordinate x of the image point is proportional to the horizontal arc length of the corresponding object point on the said object plane. The image points on the image sensor plane by all the object points on the object plane(1331) collectively form a real image.
FIG. 14 shows the cross-sectional view of the object plane in FIG. 13 in X-Z plane, and FIG. 15 shows the cross-sectional view of the object plane in FIG. 13 in Y-Z plane. From FIG. 13 through FIG. 15, the following Eq. 19 can be obtained.
                                                        A              =                            ⁢                              H                                                      tan                    ⁢                                                                                  ⁢                                          δ                      2                                                        -                                      tan                    ⁢                                                                                  ⁢                                          δ                      1                                                                                                                                              =                            ⁢                                                y                  ″                                                  tan                  ⁢                                                                          ⁢                  δ                                                                                                        =                            ⁢                                                y                  2                  ″                                                  tan                  ⁢                                                                          ⁢                                      δ                    2                                                                                                                          =                            ⁢                                                y                  1                  ″                                                  tan                  ⁢                                                                          ⁢                                      δ                    1                                                                                                                          =                            ⁢                              W                                  Δ                  ⁢                                                                          ⁢                  ψ                                                                                                        =                            ⁢                                                x                  ″                                ψ                                                                                        =                            ⁢                                                x                  1                  ″                                                  ψ                  1                                                                                                        =                            ⁢                                                x                  2                  ″                                                  ψ                  2                                                                                        [                  Equation          ⁢                                          ⁢          19                ]            
Therefore, when setting-up the size and the FOV of a desirable processed image plane, it must be ensured that Eq. 19 is satisfied.
If the processed image plane in FIG. 12 satisfies the said projection scheme, then the horizontal incidence angle of an incident ray corresponding to a lateral coordinate x″ of a third point P″ on the said processed image plane is given by Eq. 20.
                    ψ        =                                                            Δ                ⁢                                                                  ⁢                ψ                            W                        ⁢                          x              ″                                =                                    x              ″                        A                                              [                  Equation          ⁢                                          ⁢          20                ]            
Likewise, the vertical incidence angle of an incident ray corresponding to the third point having a longitudinal coordinate y″ is, from Eq. 19, given as Eq. 21.
                    δ        =                              tan                          -              1                                ⁡                      (                                          y                ″                            A                        )                                              [                  Equation          ⁢                                          ⁢          21                ]            
Therefore, the signal value of a third point on the processed image plane having an ideal projection scheme must be given as the signal value of an image point on the image sensor plane formed by an incident ray originating from an object point on the object plane having a horizontal incidence angle(i.e., the longitude) given by Eq. 20 and a vertical incidence angle(i.e., the latitude) given by Eq. 21.
According to a prior art, a panoramic image having an ideal projection scheme can be obtained as follows from a fisheye image having a distortion. First, according to the user's need, the size (W, H) of the panoramic image and the location of the third intersection point ″ are determined. The said third intersection point can be located outside the said processed image plane. In other words, the range (x″1, x″2) of the lateral coordinate and the range (y″1, y″2) of the longitudinal coordinate on the processed image plane can take arbitrary real numbers. Also, the horizontal field of view Δψ of this panoramic image (in other words, the processed image plane) is determined. Then, the horizontal incidence angle ψ and the vertical incidence angle δ of an incident ray corresponding to the rectangular coordinate (x″, y″) of a third point on the panoramic image can be obtained using Eq. 20 and Eq. 21. Next, the zenith angle θ and the azimuth angle φ of an incident ray corresponding to these horizontal and vertical incidence angles are obtained using Eq. 22 and Eq. 23.
                    ϕ        =                              tan                          -              1                                ⁡                      (                                          tan                ⁢                                                                  ⁢                δ                                            sin                ⁢                                                                  ⁢                ψ                                      )                                              [                  Equation          ⁢                                          ⁢          22                ]                                θ        =                              cos                          -              1                                ⁡                      (                          cos              ⁢                                                          ⁢              δ              ⁢                                                          ⁢              cos              ⁢                                                          ⁢              ψ                        )                                              [                  Equation          ⁢                                          ⁢          23                ]            
Next, the image height r corresponding to the zenith angle θ of the incident ray is obtained using Eq. 6. Then, using this image height r, the magnification ratio g and the azimuth angle φ of the incident ray, the two-dimensional rectangular coordinate (x′, y′) of the image point on the uncorrected image plane can be obtained as in Eq. 24 and Eq. 25.x′=gr(θ)cos φ  [Equation 24]y′=gr(θ)sin φ  [Equation 25]
In this procedure, the coordinate of the second intersection point on the uncorrected image plane, or equivalently the coordinate of the first intersection point on the image sensor plane, has to be accurately determined. Such a location of the intersection point can be easily found using various methods including image processing method. Since such techniques are well known to the people in this field, they will not be described in this document. Finally, the video signal (i.e., RGB signal) value of an image point by the fisheye lens having this rectangular coordinate is substituted as the video signal value for an image point on the panoramic image having a rectangular coordinate (x″, y″). A panoramic image having an ideal projection scheme can be obtained by image processing for all the image points on the processed image plane by the above-described method.
However, in reality, a complication arises due to the fact that all the image sensors and display devices are digital devices. Processed image plane has pixels in the form of a two-dimensional array having Jmax columns in the lateral direction and Imax rows in the longitudinal direction. Although, in general, each pixel has a square shape with both the lateral dimension and the longitudinal dimension measuring as p, the lateral and the longitudinal dimensions of a pixel are considered as 1 in the image processing field. To designate a particular pixel P″, the row number I and the column number J are used.
There is an image point—i.e., the first point—on the image sensor plane corresponding to a pixel P″ on the said processed image plane. The horizontal incidence angle of an incident ray in the world coordinate system forming an image at this first point can be written as ψI,J≡ψ(I, J). Also, the vertical incidence angle can be written as δI,J≡δ(I, J). Incidentally, the location of this first point does not generally coincide with the exact location of any one pixel.
Here, if the said processed image plane is a panoramic image, then as given by Eq. 26, the horizontal incidence angle must be a sole function of the lateral pixel coordinates J.ψI,J=ψJ≡ψ(J)  [Equation 26]
Likewise, the vertical incidence angle must be a sole function of the longitudinal pixel coordinates I.δI,J=δI≡δ(I)  [Equation 27]
Compared with the previous image processing methods, image processing methods for digitized images must follow the following set of procedures. First, the real projection scheme of the wide-angle lens that is meant to be used for image acquisition is obtained either by experiment or based on an accurate lens design prescription. Herein, when an incident ray having a zenith angle θ with respect to the optical axis forms a sharp image point on the image sensor plane by the image forming properties of the lens, the real projection scheme of the lens refers to the distance r from the intersection point  between the said image sensor plane and the optical axis to the said image point obtained as a function of the zenith angle θ of the incident ray.r=r(θ)  [Equation 28]
Said function is a monotonically increasing function of the zenith angle θ. Next, the location of the optical axis on the uncorrected image plane, in other words, the location of the second intersection point ′ on the uncorrected image plane corresponding to the first intersection point  on the image sensor plane is obtained. The pixel coordinate of this second intersection point is assumed as (Ko, Lo). In addition to this, the magnification ratio g of the pixel distance r′ on the uncorrected image plane over the real image height r on the image sensor plane is obtained. This magnification ratio g is given by Eq. 29.
                    g        =                              r            ′                    r                                    [                  Equation          ⁢                                          ⁢          29                ]            
Once such a series of preparatory stages have been completed, then a camera mounted with the said fisheye lens is installed with its optical axis aligned parallel to the ground plane, and a raw image(i.e., an uncorrected image plane) is acquired. Next, a desirable size of the processed image plane and the location (Io, Jo) of the third intersection point is determined, and then the horizontal incidence angle ψJ given by Eq. 30 and the vertical incidence angle δI given by Eq. 31 are computed for all the pixels(I, J) on the said processed image plane.
                              ψ          J                =                                                            ψ                Jmax                            -                              ψ                1                                                                    J                max                            -              1                                ⁢                      (                          J              -                              J                o                                      )                                              [                  Equation          ⁢                                          ⁢          30                ]                                          δ          I                =                              tan                          -              1                                ⁢                      {                                                                                ψ                    Jmax                                    -                                      φ                    1                                                                                        J                    max                                    -                  1                                            ⁢                              (                                  I                  -                                      I                    o                                                  )                                      }                                              [                  Equation          ⁢                                          ⁢          31                ]            
From these horizontal and vertical incidence angles, the zenith angle θI,J and the azimuth angle φI,J of the incident ray in the first rectangular coordinate system are obtained using Eq. 32 and Eq. 33.
                              θ                      I            ,            J                          =                              cos                          -              1                                ⁡                      (                          cos              ⁢                                                          ⁢                              δ                I                            ⁢              cos              ⁢                                                          ⁢                              ψ                J                                      )                                              [                  Equation          ⁢                                          ⁢          32                ]                                          ϕ                      I            ,            J                          =                              tan                          -              1                                ⁡                      (                                          tan                ⁢                                                                  ⁢                                  δ                  I                                                            sin                ⁢                                                                  ⁢                                  ψ                  J                                                      )                                              [                  Equation          ⁢                                          ⁢          33                ]            
Next, the image height rI,J on the image sensor plane is obtained using Eq. 32 and Eq. 28.rI,J=r(θI,J)  [Equation 34]
Next, using the location (Ko, Lo) of the second intersection point on the uncorrected image plane and the magnification ratio g, the location of the second point on the uncorrected image plane is obtained using Eq. 35 and Eq. 36.x′I,J=Lo+grI,J cos φI,J  [Equation 35]y′I,J=Ko+grI,J sin φI,J  [Equation 36]
The location of the said second point does not exactly coincide with the location of any one pixel. Therefore, (x′I,J, y′I,J) can be considered as the coordinate of a virtual pixel on the uncorrected image plane corresponding to the third point on the processed image plane, and they are real numbers in general.
Since the said second point does not coincide with any one pixel, an appropriate interpolation method must be used for image processing. FIG. 16 is a panoramic image following a cylindrical projection scheme that has been extracted from the image in FIG. 5, of which the lateral and the longitudinal dimensions are both 250 pixels, and the third intersection point is located at the center of the processed image plane. Furthermore, the horizontal FOV of the processed image plane is 180° (i.e., π). As can be seen from FIG. 16, all the vertical lines in the three walls, namely the front, the left, and the right walls in FIG. 5 appear as straight lines in FIG. 16.
On the other hand, FIG. 17 is an exemplary image of an interior scene, which has been acquired by aligning the optical axis of a fisheye lens with 190° FOV described in references 5 and 6 parallel to the ground plane. The real projection scheme of this fisheye lens is described in detail in the said references. On the other hand, FIG. 18 is a panoramic image having a cylindrical projection scheme extracted from the fisheye image in FIG. 17. Here, the width:height ratio of the processed image plane is 16:9, the position of the third intersection point coincides with the center of the processed image plane, and the horizontal FOV is set as 190°. As can be seen from FIG. 18, all the vertical lines are captured as vertical lines and all the objects appear natural. Slight errors are due to the error in aligning the optical axis parallel to the ground plane, and the error in experimentally determining the position of the optical axis on the uncorrected image plane.
Inventions in reference 8 provide mathematically accurate image processing algorithms for extracting panoramic images and devices implementing the algorithms. In many cases, however, distortion-free rectilinear images can be more valuable. Or, it can be more satisfactory when panoramic images and rectilinear images are both available. FIG. 19 is a conceptual drawing illustrating the rectilinear projection scheme of a prior art described in reference 9. A lens with a rectilinear projection scheme is a so-called distortion-free lens, and the characteristics of a rectilinear lens are considered identical to those of a pinhole camera. To acquire an image with such a rectilinear projection scheme, we assume an object plane(1931) and a processed image plane(1935) in the world coordinate system as schematically shown in FIG. 19.
The imaging system in this embodiment is heading in an arbitrary direction, and the third rectangular coordinate system takes the optical axis(1901) of the imaging system as the negative z″-axis, and the nodal point of the lens as the origin. Image sensor plane has a rectangular shape with a lateral width B and a longitudinal height V, and the image sensor plane is a plane perpendicular to the optical axis. On the other hand, the processed image plane has a rectangular shape with a lateral width W and a longitudinal height H. The x-axis of the first rectangular coordinate system, the x′-axis of the second rectangular coordinate system, the x″-axis of the third rectangular coordinate system and the X-axis of the world coordinate system are all parallel to the sides of the image sensor plane along the lateral direction. Furthermore, the z-axis of the first rectangular coordinate system, the z′-axis of the second rectangular coordinate system, and the z″-axis of the third rectangular coordinate systems are all identical to each other and are heading to the exact opposite direction to the Z-axis of the world coordinate system.
In this embodiment, the processed image plane is assumed to be located at a distance s″ from the nodal point of the lens. In a rectilinear projection scheme, the shape of the object plane(1931) is also a plane perpendicular to the optical axis, and the image of objects on the object plane is faithfully reproduced on the processed image plane(1935) with both the lateral and the longitudinal scales preserved. The ideal projection scheme of a rectilinear lens is identical to the projection scheme of a pinhole camera. Considering the simple geometrical characteristics of a pinhole camera, it is convenient to assume that the shape and the size of the object plane(1931) are identical to those of the processed image plane. Therefore, the distance from the object plane(1931) to the nodal point N of the lens is also assumed as s″.
FIG. 20 illustrates the case where the intersection point  between the image sensor plane and the optical axis, or equivalently the third intersection point ″ on the processed image plane corresponding to the first intersection point  does not coincide with the center C″ of the processed image plane(2035). Therefore, it corresponds to an imaging system with a slide operation as has been described in an embodiment of the prior art. In two-dimensional rectangular coordinate system having the third intersection point as the origin, the coordinate of the said center C″ is given as (x″c, y″c). Since the lateral dimension of the processed image plane is W, the lateral coordinate with respect to the center C″ has a minimum value x″1=−W/2 and a maximum value x″2=W/2. Considering the coordinate of the center C″ on top of this, the range of the lateral coordinate of the processed image plane has a minimum value x″1=x″c−W/2 and a maximum value x″2=x″c+W/2. Likewise, the range of the longitudinal coordinate has a minimum value y″1=y″c−H/2 and a maximum value y″2=y″c+H/2.
The distance between the third intersection point ″ on the processed image plane to the third point P″, in other words, the image height r″ is given by Eq. 37.r″=√{square root over ((x″)2+(y″)2)}  [Equation 37]
Since the virtual distance from the nodal point of the lens to the processed image plane is s″, an incident ray arriving at the third point by the rectilinear lens has a zenith angle given by Eq. 38.
                    θ        =                              tan                          -              1                                ⁡                      (                                          r                ″                                            s                ″                                      )                                              [                  Equation          ⁢                                          ⁢          38                ]            
On the other hand, the azimuth angle of the said incident ray is given by Eq. 39.
                    ϕ        =                              ϕ            ″                    =                                    tan                              -                1                                      ⁡                          (                                                y                  ″                                                  x                  ″                                            )                                                          [                  Equation          ⁢                                          ⁢          39                ]            
Therefore, when an incident ray having the said zenith angle and the azimuth angle forms an image point on the image sensor plane by the image forming properties of the lens, the coordinate of the image point is given by Eq. 40 and Eq. 41.x′=gr(θ)cos φ  [Equation 40]y′=gr(θ)sin φ  [Equation 41]
Therefore, it is only necessary to substitute the signal value of the third point on the processed image plane by the signal value of the image point on the uncorrected image plane having such rectangular coordinate.
Similar to the embodiment of a prior art previously described, considering the facts that all the image sensors and the display devices are digital devices, it is convenient to use the following set of equations in image processing procedure. First of all, the size (Imax, Jmax) of the processed image plane and the horizontal FOV Δψ prior to any slide operation are determined. Then, the pixel distance s″ between the nodal point of the lens and the processed image plane can be obtained using Eq. 42.
                              s          ″                =                                            J              max                        -            1                                2            ⁢                          tan              ⁡                              (                                  Δψ                  2                                )                                                                        [                  Equation          ⁢                                          ⁢          42                ]            
Furthermore, the center coordinate of the processed image plane is given by Eq. 43.
                              (                                    I              o                        ,                          J              o                                )                =                  (                                                    1                +                                  I                  max                                            2                        ,                                          1                +                                  J                  max                                            2                                )                                    [                  Equation          ⁢                                          ⁢          43                ]            
Here, Eq. 43 reflects the convention that the coordinate of the pixel on the upper left corner of a digital image is designated as (1, 1).
Next, according to the needs, the displacement (ΔI, ΔJ) of the said center from the third intersection point is determined. Once such preparatory stages have been finished, the zenith angle given in Eq. 44 and the azimuth angle given in Eq. 45 are calculated for every pixel on the processed image plane.
                              θ                      I            ,            J                          =                              tan                          -              1                                ⁢                      {                                                                                                      (                                              I                        -                                                  I                          o                                                +                                                  Δ                          ⁢                                                                                                          ⁢                          I                                                                    )                                        2                                    +                                                            (                                              J                        -                                                  J                          o                                                +                                                  Δ                          ⁢                                                                                                          ⁢                          J                                                                    )                                        2                                                                              s                ″                                      }                                              [                  Equation          ⁢                                          ⁢          44                ]                                          ϕ                      I            ,            J                          =                              tan                          -              1                                ⁡                      (                                          I                -                                  I                  o                                +                                  Δ                  ⁢                                                                          ⁢                  I                                                            J                -                                  J                  o                                +                                  Δ                  ⁢                                                                          ⁢                  J                                                      )                                              [                  Equation          ⁢                                          ⁢          45                ]            
Next, the image height rI,J on the image sensor plane is calculated using Eq. 46.rI,J=r(θI,J)  [Equation 46]
Next, the position of the second point on the uncorrected image plane is calculated using the position (Ko, Lo) of the second intersection point on the uncorrected image plane and the magnification ratio g.x′I,J=Lo+grI,J cos(φI,J)  [Equation 47]y′I,J=Ko+grI,J sin(φI,J)  [Equation 48]
Once the position of the corresponding second point has been found, the rectilinear image can be obtained using the previously described interpolation methods.
FIG. 21 is a rectilinear image extracted from the fisheye image given in FIG. 5, of which the lateral and the longitudinal dimensions are both 250 pixels, and there is no slide operation. As can be seen from FIG. 21, all the straight lines are captured as straight lines. On the other hand, FIG. 22 is a rectilinear image extracted from FIG. 17 with the width:height ratio of 4:3. The position of the third intersection point coincides with the center of the processed image plane, and the horizontal FOV is 60°. Here, it can be seen that all the straight lines in the world coordinate system are captured as straight lines in the processed image plane.
Panoramic imaging system in reference 8 requires a direction sensing means in order to provide natural-looking panoramic images at all the times irrespective of the inclination of the device having the imaging system with respect to the ground plane. However, it may happen that additional installation of a direction sensing means may be difficult in terms of cost, weight, or volume for some devices such as motorcycle or unmanned aerial vehicle. FIG. 23 is a conceptual drawing illustrating the definition of a multiple viewpoint panoramic image that can be advantageously used in such cases.
An imaging system providing multiple viewpoint panoramic images is comprised of an image acquisition means for acquiring an uncorrected image plane which is equipped with a wide-angle lens rotationally symmetric about an optical axis, an image processing means for producing a processed image plane from the uncorrected image plane, and an image display means for displaying the processed image plane on a screen with a rectangular shape.
The processed image plane in FIG. 23 is composed of three sub rectilinear image planes, namely, the 1st sub rectilinear image plane(2331-1), the 2nd sub rectilinear image plane(2331-2) and the 3rd sub rectilinear image plane(2331-3). The 1st through the 3rd sub rectilinear image planes are laid out horizontally on the processed image plane. More generally, the said processed image plane is a multiple viewpoint panoramic image, wherein the said multiple viewpoint panoramic image is comprised of the 1st through the nth sub rectilinear image planes laid out horizontally on the said screen, n is a natural number larger than 2, an arbitrary straight line in the world coordinate system having the nodal point of the wide-angle lens as the origin appears as a straight line(2381A) on any of the 1st through the nth sub rectilinear image plane, and any straight line in the world coordinate system appearing on more than two adjacent sub rectilinear image planes appears as a connected line segments(2381B-1, 2381B-2, 2381B-3).
FIG. 24 is a conceptual drawing of an object plane providing a multiple viewpoint panoramic image. Object plane of the current embodiment has a structure where more than two planar sub object planes are joined together. Although FIG. 24 is illustrated as a case where three sub object planes, namely 2431-1, 2431-2 and 2431-3 are used, a more general case of using n sub object planes can be easily understood as well. In order to easily understand the current embodiment, a sphere with a radius T centered at the nodal point N of the lens is assumed. If a folding screen is set-up around the sphere while keeping the folding screen to touch the sphere, then this folding screen corresponds to the object plane of the current embodiment. Therefore, the n sub object planes are all at the same distance T from the nodal point of the lens. As a consequence, all the sub object planes have an identical zoom ratio or a magnification ratio.
In FIG. 24 using three sub object planes, the principal direction of vision(2401-1) of the 1st sub object plane(2431-1) makes an angle of ψ1-2 with the principal direction of vision(2401-2) of the 2nd sub object plane(2431-2), and the principal direction of vision(2401-3) of the 3rd sub object plane(2431-3) makes an angle of ψ3-4 with the principal direction of vision(2401-2) of the 2nd sub object plane(2431-2). The range of the horizontal FOV of the 1st sub object plane is from a minimum value ψ1 to a maximum value ψ2, and the range of the horizontal FOV of the 2nd sub object plane is from a minimum value ψ2 to a maximum value ψ3. By having the horizontal FOVs of adjacent sub object planes be seamlessly continued, a natural looking multiple viewpoint panoramic image can be obtained. The 1st sub object plane and the 3rd object plane are obtained by panning the 2nd sub object plane by appropriate angles.
FIG. 25 is another example of a fisheye image, and it shows the effect of installing a fisheye lens with 190° FOV on the ceiling of an interior. On the other hand, FIG. 26 is a multiple viewpoint panoramic image extracted from FIG. 25. Each sub object plane has a horizontal FOV of 190°/3. From FIG. 26, it can be seen that such an imaging system will be useful as an indoor security camera.
FIG. 27 is a schematic diagram of an imaging system embodying the conception of the present invention and the invention of prior arts, and it is comprised of an image acquisition means(2710), an image processing means(2716), and an image display means(2717). The image processing means(2716) of the present invention has an input frame buffer(2771) storing one frame of image acquired from the image acquisition means(2710). The input frame buffer(2771) stores a digital image acquired from the image acquisition means(2710) in the form of a two-dimensional array. This image is the uncorrected image plane. On the other hand, the output frame buffer(2773) stores an output signal in the form of a two-dimensional array, which corresponds to a processed image plane(2735) that can be displayed on the image display means(2717). A central processing unit(2775) further exists, which generates a processed image plane from the uncorrected image plane existing in the input frame buffer and stores the processed image plane in the output frame buffer. The mapping relation between the output frame buffer and the input frame buffer is stored in a non-volatile memory(2779) such as a Flash memory in the form of a lookup table (LUT). In other words, using the algorithms from the embodiments of the current invention, a long list of pixel addresses for the input frame buffer corresponding to particular pixels in the output frame buffer is generated and stored. Central processing unit(2775) refers to this list stored in the nonvolatile memory in order to process the image.
On the other hand, an image selection device(2777) receives signals coming from various sensors and image selection means and sends them to the central processing unit. Also, by recognizing the button pressed by the user, the image selection device can dictate whether the original distorted fisheye image is displayed without any processing, or a panoramic image with a cylindrical or a Mercator projection scheme is displayed, or a rectilinear image is displayed. Said nonvolatile memory stores a number of list corresponding to the number of possible options a user can choose.
In these various cases, the wide-angle lens rotationally symmetric about an axis that is mounted on the said image acquisition means can be a refractive fisheye lens with a FOV larger than 180°, but sometimes a catadioptric fisheye lens may be needed. For the projection scheme, equidistance projection scheme can be used, but stereographic projection scheme can be used, also. Although, fisheye lenses with stereographic projection schemes are preferable in many aspects in general, fisheye lenses with stereographic projection schemes are much harder both in design and in manufacturing. Therefore, fisheye lenses with equidistance projection schemes can be realistic alternatives.
Depending on the application areas, the image display means(2717) can be a computer screen, a CCTV monitor, a CRT monitor, a digital television, an LCD projector, a network monitor, the display screen of a cellular phone, a navigation module for an automobile, and other various devices.
Such imaging systems have two drawbacks. Firstly, since image processing means needs additional components such as DSP chip or Flash memory, the manufacturing cost of the imaging system is increased. Secondly, image processing takes more than several tens of milliseconds, and a time gap exist between the image displayed on the image display means and the current state of the actual objects. Several tens of milliseconds is not a long time, but it can correspond to several meters for an automobile driving at a high speed. Therefore, application areas exist where even such a short time gap is not allowed. The gap will be greater in other application areas such as airplanes and missiles.
Reference 10 discloses a technical conception for providing panoramic images without image processing by deliberately matching the pixel locations within an image sensor to desired panoramic images. Shown in FIG. 28 is a schematic diagram of a general catadioptric panoramic imaging system. As schematically shown in FIG. 28, a catadioptric panoramic imaging system of prior arts includes as constituent elements a rotationally symmetric panoramic mirror(2811), of which the cross-sectional profile is close to an hyperbola, a lens(2812) that is located on the rotational-symmetry axis(2801) of the mirror(2811) and oriented toward the said mirror(2811), and a camera body(2814) having an image sensor(2813) inside. Then, an incident ray(2805) having a zenith angle θ, which originates from every 360° directions around the mirror and propagates toward the rotational-symmetry axis(2801), is reflected at a point M on the mirror surface(2811), and captured by the image sensor(2813). The image height is given as r=r(θ).
FIG. 29 is a conceptual drawing of an exemplary rural landscape obtainable using the catadioptric panoramic imaging system of prior art schematically shown in FIG. 28. As illustrated in FIG. 29, a photographic film or an image sensor(2813) has a square or a rectangular shape, while a panoramic image obtained using a panoramic imaging system has an annular shape. Non-hatched region in FIG. 29 constitutes a panoramic image, and the hatched circle in the center corresponds to the area at the backside of the camera, which is not captured because the camera body occludes its view. An image of the camera itself reflected by the mirror(2811) lies within this circle. On the other hand, the hatched regions at the four corners originate from the fact that the diagonal field of view of the camera lens(2812) is larger than the field of view of the panoramic mirror(2811). The image of the scene in front of the camera that is observable in absence of the panoramic mirror lies in these regions. FIG. 30 is an exemplary unwrapped panoramic image obtained from the ring-shaped panoramic image in FIG. 29 by cutting along the cutting-line and converting into a perspectively normal view using image processing software.
FIG. 31 shows geometrical relations necessary to transform the ring-shaped panoramic image illustrated in FIG. 29 into a perspectively normal unwrapped panoramic image shown in FIG. 30. Origin O of the rectangular coordinate system lies at the center of the image sensor plane(3113), the x-coordinate increases from the left side of the image sensor plane to the right side of the image sensor plane, and the y-coordinate increases from the top end of the image sensor plane to the bottom end of the image sensor plane. Image plane(3133) on the image sensor plane(3113) has a ring shape defined by a concentric inner rim(3133a) and an outer rim(3133b). The radius of the inner rim(3133a) is r1 and the radius of the outer rim(3133b) is r2. The center coordinate of the panoramic image plane(3133) in a rectangular coordinate system is (0, 0), and the coordinate of a point P on the panoramic image plane defined by the inner rim(3133a) and the outer rim(3133b) is given as (x, y). On the other hand, the coordinate of the said point P in a polar coordinate is given as (r, θ). Variables in the rectangular coordinate system and the polar coordinate system satisfy simple geometrical relations given in Eq. 49 through Eq. 52. Using these relations, variables in the two coordinate systems can be readily transformed into each other.
                    x        =                  r          ⁢                                          ⁢          cos          ⁢                                          ⁢          ϕ                                    [                  Equation          ⁢                                          ⁢          49                ]                                y        =                  r          ⁢                                          ⁢          sin          ⁢                                          ⁢          ϕ                                    [                  Equation          ⁢                                          ⁢          50                ]                                r        =                                            x              2                        +                          y              2                                                          [                  Equation          ⁢                                          ⁢          51                ]                                ϕ        =                              tan                          -              1                                ⁡                      (                          y              x                        )                                              [                  Equation          ⁢                                          ⁢          52                ]            
Here, the lateral coordinate on the unwrapped panoramic image is proportional to the azimuth angle φ, and the longitudinal coordinate is proportional to radius r.
As described previously, reference 10 discloses a CMOS image sensor which does not require image processing in order to transform ring-shaped panoramic image into a rectangular unwrapped panoramic image.
As schematically shown in FIG. 32, ordinary CMOS image sensor plane(3213) is comprised of multitude of pixels(3281) arranged in a matrix form, a vertical shift register(3285), and a horizontal shift register(3284). Each pixel(3281) is comprised of a photodiode(3282) converting the received light into electrical charges proportional to its light intensity, and a photoelectrical amplifier(3283) converting the electrical charges (i.e., the number of electrons) into an electrical voltage. The light captured by a pixel is converted into an electrical voltage at the pixel level, and outputted into a signal line by the vertical and the horizontal shift registers. Here, the vertical shift register can selectively choose among the pixels belonging to different rows, and the horizontal shift register can selectively choose among the pixels belonging to different columns. A CMOS image sensor having such a structure operates in a single low voltage, and consumes less electrical power.
A photoelectrical amplifier(3283) existing in each pixel(3281) occupies a certain area which does not contribute to light capturing. The ratio of the light receiving area(i.e., the photodiode) over the pixel area is called the Fill Factor, and if the Fill Factor is large, then the light detection efficiency is high.
A CMOS image sensor having such structural characteristics has a higher level of freedom in design and pixel arrangement compared to a CCD image sensor. FIG. 33 is a floor plan showing pixel arrangement in a CMOS image sensor for panoramic imaging system according to an embodiment of the invention of a prior art. The CMOS image sensor has an imaging plane(3313) whereon pixels(3381) are arranged. Although pixels(3381) in FIG. 33 have been represented as dots, these dots are merely the center positions representing the positions of the pixels, and actual pixels(3381) occupy finite areas around the said dots. Each pixel(3381) is positioned on intersections between M radial lines(3305) having a position O on the image sensor plane as a starting point and N concentric circles(3306) having the said one point O as the common center. M and N are both natural numbers, and the radius rn of the nth concentric circle from the center position O is given by Eq. 53.
                              r          n                =                                                            n                -                1                                            N                -                1                                      ⁢                          (                                                r                  N                                -                                  r                  1                                            )                                +                      r            1                                              [                  Equation          ⁢                                          ⁢          53                ]            
In Eq. 53, n is a natural number ranging from 1 to N, r1 is the radius of the first concentric circle from the center position O, and rn is the radius of the nth concentric circle from the center position O. All the M radial lines maintain an identical angular distance. In other words, pixels on a given concentric circle are positioned along the perimeter of the concentric circle maintaining an identical interval. Furthermore, by Eq. 53, each pixel is positioned along a radial line maintaining an identical interval.
Considering the characteristics of panoramic images, it is better if the image sensor plane(3313) has a square shape. Preferably, each side of an image sensor plane(3313) is not smaller than 2rN, and the center O of the concentric circles is located at the center of the image sensor plane(3313).
A CMOS image sensor for panoramic imaging system illustrated in FIG. 33 have pixels(3381) arranged at regular interval along the perimeters of concentric circles 3306), wherein the concentric circles have a regular interval along the radial direction, in turn. Although not schematically illustrated in FIG. 33, such a CMOS image sensor further has a radial shift register or r-register and a circumferential shift register or Φ-register. Pixels belonging to different concentric circles can be selected using the radial shift register, and pixels belonging to different radial lines on a concentric circle can be selected using the circumferential shift register.
Every pixel on the previously described CMOS image sensor according to an embodiment of the invention of a prior art has a photodiode with an identical size. On the other hand, pixel's illuminance decreases from the center of the CMOS image sensor plane(3313) toward the boundary according to the well-known cosine fourth law. Cosine fourth law becomes an important factor when an exposure time has to be determined for a camera lens with a wide field-of-view, or the image brightness ratio between the pixel located at the center of the image sensor plane and the pixel at the boundary has to be determined. FIG. 34 shows the structure of a CMOS image sensor for panoramic imaging system for resolving the illuminance difference between the center and the boundary of an image sensor plane according to another embodiment of the invention of a prior art. Identical to the embodiment previously described, each pixel is positioned at a regular interval on radial lines, wherein the radial lines in turn have an identical angular distance on a CMOS image sensor plane(3413). In other words, each pixel is positioned along the perimeter of a concentric circle at a regular interval, and the concentric circles on the image sensor plane have an identical interval along the radial lines. Although pixels in FIG. 34 have been represented as dots, these dots merely represent the center positions of pixels, and actual pixels occupy finite areas around these dots.
Since all the pixels on the image sensor plane have an identical angular distance along the perimeter, the maximum area on the image sensor plane which a particular pixel can occupy increases proportionally to the radius. In other words, a pixel(3430j) existing on a circle on the image sensor plane with a radius rj can occupy a larger area than a pixel(3430i) existing on a circle with a radius ri (wherein, ri<rj). Since pixels located far from the center of the image sensor plane can occupy large areas than pixels located near the center, if each pixel's photoelectrical amplifier is identical in size, then pixels near the boundary of the image sensor plane far from the center can have relatively larger photodiodes. By having a larger light receiving area, the fill factor can be increased, and the decrease in illuminance by the cosine fourth law can be compensated. In this case, the size of the photodiode can be proportional to the radius of the concentric circle.
On the other hand, reference 11 discloses a structure of a CMOS image sensor for compensating the lens brightness decreasing as it moves away from the optical axis by making the photodiode area becomes larger as it moves away from the optical axis.    [reference 1] J. F. Blinn and M. E. Newell, “Texture and reflection in computer generated images”, Communications of the ACM, 19, 542-547 (1976).    [reference 2] N. Greene, “Environment mapping and other applications of world projections”, IEEE Computer Graphics and Applications, 6, 21-29 (1986).    [reference 3] E. W. Weisstein, “Cylindrical Projection”, http://mathworld.wolfram.com/CylindricalProjection.html.    [reference 4] W. D. G. Cox, “An introduction to the theory of perspective—part 1”, The British Journal of Photography, 4, 628-634 (1969).    [reference 5] G. Kweon and M. Laikin, “Fisheye lens”, Korean patent 10-0888922, date of patent Mar. 10, 2009.    [reference 6] G. Kweon, Y. Choi, and M. Laikin, “Fisheye lens for image processing applications”, J. of the Optical Society of Korea, 12, 79-87 (2008).    [reference 7] G. Kweon and M. Laikin, “Wide-angle lenses”, Korean patent 10-0826571, date of patent Apr. 24, 2008.    [reference 8] G. Kweon, “Methods of obtaining panoramic images using rotationally symmetric wide-angle lenses and devices thereof”, Korean patent 10-0882011, date of patent Jan. 29, 2009.    [reference 9] G. Kweon, “Method and apparatus for obtaining panoramic and rectilinear images using rotationally symmetric wide-angle lens”, Korean patent 10-0898824, date of patent May 14, 2009.    [reference 10] G. Kweon, “CMOS image sensor and panoramic imaging system having the same”, Korean patent 10-0624051, date of patent Sep. 7, 2006.    [reference 11] A. Silverstein, “Method, apparatus, and system providing a rectilinear pixel grid with radially scaled pixels”, international application number PCT/US2008/060185, date of international application Apr. 14, 2008.